A Concise Information-Theoretic Derivation of the Baum-Welch algorithm

We derive the Baum-Welch algorithm for hidden Markov models (HMMs) through an information-theoretical approach using cross-entropy instead of the Lagrange multiplier approach which is universal in machine learning literature. The proposed approach provides a more concise derivation of the Baum-Welch method and naturally generalizes to multiple observations. Introduction The basic hidden Markov model (HMM)[5] is defined as having a sequence of hidden or latent states Q = {qt} = {q1, q2, . . . , qT } (where t denotes time interval), and each state is statistically independent of all but the state immediately before it and where each state emits some observation ot with a stationary (non-time-varying) probability density. Formally, the model is defined as: p(O,Q|λ) = p(q1|λ) [ T ∏ t=2 p(qt|qt−1, λ) ] [ T ∏